Method and device for implementing frame synchronization by controlling transit time

ABSTRACT

Present Invention provides a method of virtual reality image transition configured to implement frame synchronization by controlling transit time among image segmentation expressing virtual reality comprises, (a) determining a key frame distance (KFD) of image segmentation expressing a part of original image for implementing virtual reality, (b) implementing frame synchronization between image segmentation before and after transition by executing a transition of image segmentation only in a key frame among frames composing image segmentation, when reproduced image segmentation is transited as the user&#39;s gaze changes.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a US Bypass Continuation Application ofInternational Application No. PCT/KR2020/001517, filed on Jan. 31, 2020and designating the United States, the International Applicationclaiming a priority date of Jan. 31, 2019 based on prior Korean PatentApplication No. 10-2019-0012531, filed on Jan. 31, 2019. The disclosureof which is incorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Invention—Technical Field

The present invention relates to a transit method of virtual realityimage segmentation and a virtual reality image reproducing device.

2. Discussion of Related Art

When the original image that implements virtual reality is an imagecovers 360°, it is unnecessary to reproduce the part of the image thatthe user's gaze does not reach.

Therefore, after dividing into a plurality of image segmentation bydividing the original image into a predetermined segment, reproducingonly the image segmentation corresponding to the part of the user's gazehelps to use the hardware efficiently.

However, if the image segmentation is reproduced in this manner, theimage segmentation that is reproduced should be transited as the user'sgaze moves. However, the screen freeze problem occurs if the transittime is delayed too much.

In addition, the problem that the previously reproduced image isreplayed or a scene to be reproduced is skipped occurs, if the frames ofimage segmentation before transition and image segmentation aftertransition are not synchronized.

Accordingly, the inventor of the present invention has completed thepresent invention after a long period of research and trial and error inorder to solve such a problem.

SUMMARY OF THE INVENTION

The present invention, in order to solve the above-described problems,provides a virtual reality image segmentation transit method and avirtual reality image reproducing device that controls the transit timeof image segmentation reproduced among a plurality of image segmentationto prevent screen freeze.

In addition, the present invention provides the virtual reality imagesegmentation transit method and the virtual reality image reproducingdevice that synchronize frames of image segmentation before transitionand image segmentation after transition.

The technical problems to be achieved in the present invention are notlimited to the above technical problems, and other technical problemsnot mentioned will be clearly understood by those of ordinary skill inthe art to which the present invention belongs from the followingdescription.

One aspect of the present invention for solving the above problemsprovides a method of virtual reality image transition configured toimplement frame synchronization by controlling transit time among imagesegmentation expressing virtual reality comprising: (a) determining akey frame distance (KFD) of image segmentation expressing a part oforiginal image for implementing virtual reality; and (b) implementingframe synchronization between image segmentation before and aftertransition by executing a transition of image segmentation only in a keyframe among frames composing image segmentation, when reproduced imagesegmentation is transited as the user's gaze changes;

In a preferred embodiment, the image segmentation of step (a) isconfigured to shorten the key frame distance by not includingBidirectional frame in the image segmentation.

In a preferred embodiment, the step (a) further comprises, a step ofdetermining a Play FOV of image segmentation by adding a Guard FOV to aTemporary FOV covered by a temporary image segmentation, in thetemporary image segmentation that divides the original image into Nthregions, the step (b) starts transition procedure of image segmentationwhen the user's gaze reaches a specific point among the guard FOV ofreproduced image segmentation and prevents the user's gaze locatedoutside of the Play FOV of reproduced image segmentation before thetransition of image segmentation is finished.

In a preferred embodiment, the step (a) further comprises a step ofdetermining a Virtual Device FOV wider than a Device FOV of thehead-mounted display by a predetermined Additional FOV, the step (b)starts transition procedure of image segmentation when the user's gazereaches a specific point among the guard FOV of reproduced imagesegmentation and prevents the user's gaze located outside of the PlayFOV of reproduced image segmentation before the transition of imagesegmentation is finished.

In a preferred embodiment, the step (a) further comprises, a step ofdetermining a Play FOV of image segmentation by adding a Guard FOV to aTemporary FOV covered by a temporary image segmentation, in thetemporary image segmentation that divides the original image into Nthregions, and a step of determining a Virtual Device FOV wider than aDevice FOV of the head-mounted display by a predetermined AdditionalFOV, the step (b) comprises starting transition procedure of imagesegmentation when one end of the Virtual Device FOV reaches a specificpoint in the Guard FOV included in the Play FOV.

Another aspect of the present invention provides an image reproducerthat reproduces a segmented image that expresses a part of the originalimage for expressing virtual reality, an image segmentationsynchronization module that controls the transition time of a segmentedimage to synchronize the frames of the segmented image before and aftertransition when reproduced image segmentation is transited as the user'sgaze changes, wherein the image segmentation synchronization moduleimplements frame synchronization between image segmentation before andafter transition by executing a transition of image segmentation only ina key frame among frames composing image segmentation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A to 1C is a view for explaining a Play FOV and a Virtual DeviceFOV of the present invention.

FIG. 2 is a diagram for explaining a preferred embodiment of the VirtualReality Image segmentation transit method of the present invention.

FIG. 3 is a diagram for explaining a method of controlling the transittime of image segmentation in the present invention.

FIG. 4 is a diagram for explaining a method of synchronizing frames ofimage segmentation before and after conversion in the present invention.

It is revealed that the accompanying drawings are exemplified as areference for understanding the technical idea of the present invention,and the scope of the present invention is not limited thereby.

DETAILED DESCRIPTION

In the description of the present invention, if it is determined thatthe subject matter of the present invention is unnecessarily obscured asit is obvious to those skilled in the art with respect to related knownfunctions, the detailed description will be omitted.

FIG. 1A to 1C is a view for explaining a Play FOV and a Virtual DeviceFOV of the present invention.

Referring to FIG. 1A, temporary image segmentation is an image when theoriginal image is divided into n regions. Temporary FOV is the field ofview covered by temporary image segmentation. Temporary imagesegmentation has no overlapping regions.

Referring to FIG. 1B, Guard FOV refers to the field of view added toboth sides of the Temporary FOV to extend the Temporary FOV. Play FOV isthe field of view covered by image segmentation, and is the sum of theTemporary FOV and the Guard FOV. A plurality of image segmentationoverlaps each other by an extended Guard FOV.

Referring to FIG. 1C, Device FOV refers to a unique field of view that adevice such as a head-mounted display (HMD) can cover as a hardware.Additional FOV is a value added to the device field of view. The VirtualDevice FOV is the sum of the Device FOV and Additional FOV, and means afield of view virtually wider than the Device FOV.

FIG. 2 is a diagram for explaining a preferred embodiment of the VirtualReality Image segmentation transit method of the present invention.

The virtual reality image segmentation transit method of the presentinvention may be implemented in an image segmentation reproducingdevice. The image segmentation reproducing device is a computing deviceand may be the same device as the virtual reality image reproducingdevice or be a different device.

Referring to FIG. 2, in operation S1100, Temporary FOV which is thefield of view covered by the temporary image segmentation can becalculated by determining how many (N) temporary image segmentation isdivided from the original image. If the original image is 360° and N is4, the temporary FOV becomes 90°.

In operation S1200, the size of the Guard FOV is decided for extendingthe temporary FOV. In operation S1300, the Play FOV is determined byadding a Guard FOV to both sides of the Temporary FOV. In operationS1400, an image segmentation is generated to cover regions as much asPlay FOV. When image segmentation is generated in above manner, aplurality of image segmentation overlaps each other by the extendedGuard FOV. In a preferred embodiment, the Guard FOV is set to ½ or moreof the Device FOV of the head-mounted display.

In operation S2100, Device FOV, which is the unique field of view of adevice such as a head-mounted display (HMD), is determined. In operationS2200, Additional FOV for extending the Device FOV is determined. Inoperation S2300, Virtual Device FOV is determined by adding AdditionalFOV to Device FOV.

In operation S3100, The Virtual Device FOV that varies according to theuser's gaze is compared with the Play FOV of the image segmentationbeing reproduced.

In operation S3200, a transit procedure of image segmentation is startedwhen a predetermined transit standard calculated using the VirtualDevice FOV and the Play FOV of the image segmentation being reproducedis satisfied.

A step of implementing frame synchronization, between image segmentationbefore and after the transition by executing the transition of imagesegmentation only in the key frame of image segmentation, may becomprised when the transit procedure of image segmentation is started

In a preferred embodiment, the transit standard may be determined thatthe reproduced image segmentation among the plurality of imagesegmentation is transited when one end of the Virtual Device FOV reachesa specific point in the Guard FOV included in the Play FOV. The specificpoint is determined within the range of the Guard FOV.

In another embodiment, the transit standard may be determined that thereproduced image segmentation using the reference angle θL of theVirtual Device FOV is transited. The reference angle will be describedin detail with reference to FIG. 3.

FIG. 3 is a diagram for explaining a method of controlling the transittime of image segmentation in the present invention.

Referring to FIG. 3, the reference angle θL of the Virtual Device FOVdetermined by the equation below. The reference angle θL is the left endof the Virtual Device FOV. The angle is positive in the counterclockwisedirection.

θL(k)∈[−(k+½)θN+Δθθm,−(k−½)θN+θg+θm]

N is the number of image segmentation, k is an integer from 0 to N−1,Δθ=θf−θg, θN=field of view of original image/N, θg is a Guard FOV, Of isa Virtual Device FOV, θm is an additional angle to locate object intothe center of user's gaze.

An embodiment of the present invention determines that which imagesegmentation is the image segmentation belongs to the reference angle,by using the reference angle θL of the Virtual Device FOV determined bythe above equation. And, if it is determined that the image segmentationbelongs to reference angle has been changed, the reproduced imagesegmentation is transited.

A virtual reality image reproducing device configured to control transittime of reproduced image among a plurality of image segmentationreproduce image segmentation by using the above method. In a preferredembodiment, the virtual reality image reproducing device may include animage reproducer, a virtual FOV generator, and a reproducing imagetransit module.

The image reproducer reproduces image segmentation that expresses a partof the original image for implementing virtual reality. The virtual FOVgenerator generates a Virtual Device FOV that is wider than the DeviceFOV of the head-mounted display by a predetermined Additional FOV.

The reproducing image transit module determines a transition ofreproduced image segmentation among a plurality of image segmentationwhen one end of the Virtual Device FOV reaches a specific point in theGuard FOV included in the Play FOV.

The Play FOV of the image segmentation is generated by adding a GuardFOV to a Temporary FOV covered by the temporary image segmentation thatdivides the original image into Nth regions, which is as describedabove.

[Frame Synchronization of Image Segmentation Before and afterTransition]

A group of pictures (GOP) refers to a set including at least one I framein MPEG image processing. Frames include I (Intra-coded), P(Predictive-coded), and B (Bidirectional-coded) frames. I frames arekeyframes and are encoded without reference to other frames.Accordingly, the I frames can be independently decoded without referenceto other frames.

P frames are coded with reference to the previous I frames and P frames.B frames are coded using both the previous and subsequent I frames and Pframes. Accordingly, it takes a considerable amount of time to decodethe B frames.

Therefore, when the frame of image segmentation reproduced aftertransition is frame B, the time required for decoding may greatlyincrease, which may lead to synchronization failure.

Accordingly, the present invention maintains the time required fordecoding within a predictable range by transiting between imagesegmentation only in keyframes.

In this sense, a preferred embodiment of the present invention shortensthe key frame distance (KFD) of image segmentation by using apredetermined method. And, when transiting the image segmentation thatis reproduced as the user's gaze changes, frame synchronization betweenimage segmentation before and after transition is implemented, byexecuting a transition of image segmentation only in key frame amongframes composing image segmentation.

In one embodiment, the key frame distance may be shortened by a methodof not including B frames in the image set. Since the time required fordecoding the B frames is considerable.

On the other hand, in order to perform a transition only at a key framerather than an immediate transition, a predetermined delay time must besecured. In other words, the transition should be delayed by apredetermined delay time, and there should be no problem that the screenreproduced during the delay time is cut off. To this end, the presentinvention secures the delay time by the method described below.

FIG. 4 is a diagram for explaining a method of synchronizing frames ofimage segmentation before and after conversion in the present invention.FIG. 4 illustrates a case of transiting from image segmentation Vi to Vjas an example. Tf,m is a time of the mth keyframe, Tf,m+1 is a time ofthe m+1th keyframe. Δtseek is a search time, which is the time it takesto find the next keyframe. Δtwait is a waiting time, which is the timeto continue reproducing image segmentation before transition afterstarting the transit procedure. As described above, since imagesegmentation overlaps as many as predetermined regions, the screen doesnot freeze even if the image segmentation continues to be reproducedbefore conversion to a certain extent.

Δt is the delay time, which is the sum of Δtwait and Δttransit.Δttransit is the transition time, which is the time required totransition the frame. For example, it may include the physical time toread a video file, decode frames, and prepare for display.

The delay time Δt is the time that must be secured to transit the imagesegmentation that is reproduced only in keyframes. The delay time is setto have the following range.

T_GOP>Δt=Δtwait+Δttransit>Δtseek

In other words, the delay time should be greater than the time to findthe next keyframe of image segmentation after playback. Since thetransition should be prevented when the keyframe is not found. Also, thedelay time should be smaller than T_GOP.

In addition, the delay time is set to satisfy the following range.

T_overlap>Δt

T_overlap is the overlap time, and is the time secured through the GuardFOV and Virtual Device FOV. The delay time should be less than theoverlap time. If the overlap time is less than the delay time, the videosync will be lost.

T_overlap=T_by_guard_FOV+T_by_virtual_FOV

T_by_guard_FOV is a time secured by the Guard FOV, and T_by_virtual_FOVis a time secured by the Virtual Device FOV.

If T_overlap is expressed in terms of angle and head rotation speed,T_overlap expressed as follows.

T_overlap=(2*Ooverlap+Δθdev)/head angular speed

The head rotation speed cannot be preset. The maximum head rotationspeed can be known in advance.

The stable delay time can be secured when a condition of “maximum headrotation speed(omega)<(2*theta_overlap+dtheta_dev)/{Max(t_seek)+dt_transit)}” issatisfied. Usually, each setting can be obtained by statisticallymeasuring the rotational speed of a normal person's head.

The virtual reality image reproducing device that implements framesynchronization by controlling the transit time among image segmentationexpressing virtual reality of the present invention implements framesynchronization through the above method.

In a preferred embodiment, the virtual reality image reproducing devicemay include an image reproducer and an image segmentation synchronizemodule.

The image reproducer reproduces image segmentation that expresses a partof the original image for implementing virtual reality.

The image segmentation synchronize module controls the transit time ofimage segmentation to synchronize frames of image segmentation beforeand after transition when transiting reproduced image segmentation asthe user's gaze changes.

The image segmentation synchronize module is characterized in that itimplements frame synchronization between image segmentation before andafter transition by allowing the transition of image segmentation to beexecuted only in the key frame of image segmentation.

The present invention has an effect of allowing the screen to bereproduced without screen freeze by controlling the transit time ofimage segmentation reproduced among a plurality of image segmentation.

In addition, the present invention has an effect of preventing a problemthat the previously played image is replayed or a scene to be reproducedis skipped by synchronizing frames of image segmentation beforetransition and image segmentation after transition.

On the other hand, even if it is an effect not explicitly mentionedherein, it is added that the effects described in the followingspecification expected by the technical features of the presentinvention and their potential effects are treated as described in thespecification of the present invention.

It is apparent to those skilled in the art that the present inventionmay be embodied in other specific forms without departing from thecharacteristics of the present invention. Accordingly, the abovedetailed description should not be construed as restrictive in allrespects but as exemplary. The scope of the present invention should bedetermined by a reasonable interpretation of the appended claims, andall modifications within the equivalent scope of the present inventionare included in the scope of the present invention.

What is claimed is:
 1. A method of virtual reality image transitionconfigured to implement frame synchronization by controlling transittime among image segmentation expressing virtual reality comprising: (a)determining a key frame distance (KFD) of image segmentation expressinga part of original image for implementing virtual reality; and (b)implementing frame synchronization between image segmentation before andafter transition by executing a transition of image segmentation only ina key frame among frames composing image segmentation, when reproducedimage segmentation is transited as the user's gaze changes; wherein theimage segmentation of step (a) is configured to shorten the key framedistance by not including Bidirectional frame in the image segmentation.2. A method of virtual reality image transition configured to implementframe synchronization by controlling transit time among imagesegmentation expressing virtual reality comprising: (a) determining akey frame distance (KFD) of image segmentation expressing a part oforiginal image for implementing virtual reality; and (b) implementingframe synchronization between image segmentation before and aftertransition by executing a transition of image segmentation only in a keyframe among frames composing image segmentation when reproduced imagesegmentation is transited as the user's gaze changes; wherein the step(a) further comprises, a step of determining a Play FOV of imagesegmentation by adding a Guard FOV to a Temporary FOV covered by atemporary image segmentation, in the temporary image segmentation thatdivides the original image into Nth regions, wherein the step (b) startstransition procedure of image segmentation when the user's gaze reachesa specific point among the guard FOV of reproduced image segmentationand prevents the user's gaze located outside of the Play FOV ofreproduced image segmentation before the transition of imagesegmentation is finished.
 3. A method of virtual reality imagetransition configured to implement frame synchronization by controllingtransit time among image segmentation expressing virtual realitycomprising: (a) determining a key frame distance (KFD) of imagesegmentation expressing a part of original image for implementingvirtual reality; and (b) implementing frame synchronization betweenimage segmentation before and after transition by executing a transitionof image segmentation only in a key frame among frames composing imagesegmentation when reproduced image segmentation is transited as theuser's gaze changes; wherein the step (a) further comprises a step ofdetermining a Virtual Device FOV wider than a Device FOV of thehead-mounted display by a predetermined Additional FOV, wherein the step(b) starts transition procedure of image segmentation when the user'sgaze reaches a specific point among a guard FOV of reproduced imagesegmentation and prevents the user's gaze locating outside of a Play FOVof reproduced image segmentation before the transition of imagesegmentation is finished.
 4. A method of virtual reality imagetransition configured to implement frame synchronization by controllingtransit time among image segmentation expressing virtual realitycomprising: (a) determining a key frame distance (KFD) of imagesegmentation expressing a part of original image for implementingvirtual reality; and (b) implementing frame synchronization betweenimage segmentation before and after transition by executing a transitionof image segmentation only in a key frame among frames composing imagesegmentation when reproduced image segmentation is transited as theuser's gaze changes; wherein the step (a) further comprises a step ofdetermining a Play FOV of the image segmentation by adding a Guard FOVto a Temporary FOV covered by the temporary image segmentation, in thetemporary image segmentation that divides the original image into Nthregions; and a step of determining a Virtual Device FOV wider than aDevice FOV of the head-mounted display by a predetermined AdditionalFOV, wherein the step (b) starts transition procedure of imagesegmentation when one end of the Virtual Device FOV reaches a specificpoint in the Guard FOV included in the Play FOV.